STATUS  OF  YEAR  2000  COMPLIANCE  AT  THE 
DEFENSE  COMMISSARY  AGENCY 


Report  No.  99-179  June  7,  1999 

Office  of  the  Inspector  General 
Department  of  Defense 

DISTRIBUTION  STATEMENT  A 

Approved  for  Public  Release 
Distribution  Unlimited 


19990806  045 


Ate 


Additional  Copies 


To  obtain  additional  copies  of  this  audit  report,  contact  the  Secondary  Reports 
Distribution  Unit  of  the  Audit  Follow-up  and  Technical  Support  Directorate  at 
(703)  604-8937  (DSN  664-8937)  or  fax  (703)  604-8932  or  visit  the  Inspector 
General,  DoD,  Home  Page  at:  www.dodig.osd.mil. 

Suggestions  for  Future  Audits 

To  suggest  ideas  for  or  to  request  future  audits,  contact  the  Audit  Follow-up  and 
Technical  Support  Directorate  at  (703)  604-8940  (DSN  664-8940)  or 
fax  (703)  604-8932.  Ideas  and  requests  can  also  be  mailed  to: 

OAIG-AUD  (ATTN:  AFTS  Audit  Suggestions) 

Inspector  General,  Department  of  Defense 
400  Army  Navy  Drive  (Room  801) 

Arlington,  VA  22202-2884 

Defense  Hotline 

To  report  fraud,  waste,  or  abuse,  contact  the  Defense  Hotline  by  calling 
(800)  424-9098;  by  sending  an  electronic  message  to  Hotline@dodig.osd.mil;  or 
by  writing  to  the  Defense  Hotline,  The  Pentagon,  Washington,  D.C.  20301-1900. 
The  identity  of  each  writer  and  caller  is  fully  protected. 


Acronym 

DeCA  Defense  Commissary  Agency 


INSPECTOR  GENERAL 
DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-2884 


June  7,  1999 


MEMORANDUM  FOR  DIRECTOR,  DEFENSE  COMMISSARY  AGENCY 

SUBJECT:  Audit  Report  on  Status  of  Year  2000  Compliance  at  the  Defense 
Commissary  Agency  (Report  No.  99-179) 


We  are  providing  this  report  for  information  and  use.  We  considered 
management  comments  on  a  draft  of  this  report  when  preparing  the  final  report. 

The  Defense  Commissary  Agency  comments  conformed  to  the  requirements  of 
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(csantoni@dodig.osd.mil)  or  Mr.  Robert  L.  Shaffer  at  (703)  604-9043  (DSN  664-9043) 
(rshaffer@dodig.osd.mil).  See  Appendix  B  for  the  report  distribution.  The  audit  team 
members  are  listed  inside  the  back  cover. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
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Report  No.  99-179  June  7,  1999 

(Project  No.  9AL-0083) 

Status  of  Year  2000  Compliance  at  the  Defense  Commissary  Agency 


Executive  Summary 


Introduction.  This  report  is  one  in  a  series  being  issued  by  the  Inspector  General, 
DoD,  in  accordance  with  an  informal  partnership  with  the  Chief  Information  Officer, 
DoD,  to  monitor  DoD  efforts  to  address  the  year  2000  computing  challenge.  The 
Deputy  Secretary  of  Defense  requested  this  audit  of  the  Defense  Commissary  Agency. 

Objectives.  Our  objective  was  to  evaluate  whether  the  Defense  Commissary  Agency 
was  adequately  planning  for  and  managing  year  2000  risks  to  avoid  undue  disruption  to 
its  mission.  Specifically,  we  reviewed  year  2000  risk  assessments,  testing,  and 
contingency  plans  for  Defense  Commissary  Agency  mission-critical  systems  and 
infrastructure  equipment. 

Results.  The  Defense  Commissary  Agency  was  making  progress  in  its  year  2000 
conversion  effort,  but  more  could  be  done  to  minimize  the  risk.  The  Defense 
Commissary  Agency  target  certification  date  of  October  31,  1999,  for  the  Defense 
Commissary  Agency  Interim  Business  System  may  not  allow  sufficient  time  to  correct 
unexpected  errors,  changes,  and  delays  in  solving  year  2000  issues.  In  addition,  as  of 
April  9,  1999,  the  Defense  Commissary  Agency  had  identified  the  compliance  status  of 
only  1,822  (55  percent)  of  3,285  vendor  models  of  infrastructure  equipment  that  are 
critical  to  the  operation  of  the  individual  commissaries  and  administrative  offices. 

Also,  the  contingency  plan  for  the  Defense  Commissary  Agency  Interim  Business 
System  was  inadequate,  and  the  contingency  plan  for  the  infrastructure  equipment  was 
incomplete.  As  a  result,  the  Defense  Commissary  Agency  increased  its  risk.  For 
details  of  the  audit  results,  see  the  Finding  section  of  the  report. 

Summary  of  Recommendations.  We  recommend  that  the  Director,  Defense 
Commissary  Agency,  expand  contingency  plans  to  delineate  alternative  operational 
procedures  if  the  DeCA  Interim  Business  System  or  infrastructure  equipment 
experience  system  failures  as  a  result  of  the  year  2000.  We  also  recommend  that  the 
contingency  plans  be  tested  to  ensure  that  the  alternative  procedures  are  realistic  and 
executable. 

Management  Comments.  The  Executive  Director  for  Support,  Defense  Commissary 
Agency  (the  Executive  Director)  concurred  and  stated  that  the  Defense  Commissary 
Agency  has  expanded  the  contingency  plans  to  delineate  alternative  operational 
procedures  if  the  Defense  Commissary  Agency  Interim  Business  System  and 
infrastructure  equipment  experience  system  failures  because  of  year  2000.  The  Defense 
Commissary  Agency  is  in  the  process  of  developing  a  handbook  on  contingency  plans 


for  the  infrastructure  equipment  within  the  commissaries.  Also,  the  Defense 
Commissary  Agency  is  currently  testing  or  planning  to  test  the  contingency  plans  to 
ensure  the  alternative  procedures  are  realistic  and  executable.  The  Executive  Director 
stated  that  the  Defense  Commissary  Agency  has  made  significant  progress  in  lowering 
the  year  2000  risk  associated  with  evaluating  the  compliance  status  of  infrastructure 
equipment  that  are  critical  to  the  operation  of  the  individual  commissaries  and 
administrative  offices.  As  of  May  10,  1999,  the  Defense  Commissary  Agency 
identified  the  compliance  status  of  79  percent  of  the  vendor  models  of  infrastructure 
equipment.  The  target  completion  date  for  all  ongoing  tasks  is  October  1999.  See  the 
Finding  section  of  the  report  for  a  discussion  of  the  management  comments  and  the 
Management  Comment  section  of  the  report  for  a  complete  text  of  the  comments. 
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Background 


Executive  Order  No.  13073.  Because  of  the  potential  failure  of  computers  to 
run  or  function  throughout  the  Government,  the  President  issued  Executive 
Order  No.  13073,  “Year  2000  Conversion,”  February  4,  1998.  The  executive 
order  makes  it  policy  that  Federal  agencies  ensure  that  no  critical  Federal 
program  experiences  disruption  because  of  the  year  2000  computing  problem. 
The  Executive  Order  also  requires  the  head  of  each  agency  to  ensure  that  efforts 
to  address  the  year  2000  computing  problem  receive  the  highest  priority. 

DoD  Year  2000  Management  Plan.  The  DoD  Year  2000  Management  Plan, 
(DoD  Management  Plan)  provides  the  overall  DoD  strategy  and  guidance  to 
inventory,  prioritize,  fix,  or  retire  systems,  and  to  monitor  their  progress.  DoD 
Components  are  responsible  for  implementing  the  five-phase,  year  2000 
management  process  described  in  the  DoD  Management  Plan.  The  target  date 
to  make  mission-critical  systems  year  2000  compliant  was  December  31,  1998. 
Systems  that  were  still  noncompliant  after  that  date  were  to  be  considered  high 
risk.  The  Office  of  Management  and  Budget  target  date  was  March  31 ,  1999. 

Defense  Commissary  Agency.  The  Defense  Commissary  Agency  (DeCA) 
mission  is  to  provide  an  efficient  and  effective  worldwide  system  of 
commissaries  for  the  resale  of  groceries  and  household  supplies  at  the  lowest 
practical  price  to  members  of  the  military  Services,  their  families,  and  other 
authorized  patrons,  while  maintaining  high  standards  for  the  quality  of  facilities, 
products,  and  services.  DeCA  reports  to  the  Under  Secretary  of  Defense  for 
Personnel  and  Readiness.  Part  of  the  DeCA  mission  is  to  promote  positive 
morale  among  its  patrons  in  support  of  personnel  retention  and  combat 
readiness.  The  DeCA  operates  approximately  294  commissaries  in  4  regional 
areas  worldwide.  To  operate  the  294  commissaries,  DeCA  relies  heavily  on  a 
network  of  business  systems. 

The  DeCA  designated  four  mission-critical  business  systems  whose  functions 
are  required  to  ensure  that  the  groceries  can  be  purchased  and  sold,  shelves  can 
be  stocked,  and  vendors  can  be  paid.  Specifically,  the  four  mission-critical 
systems  are  the  DeCA  Interim  Business  System,  the  Point  of  Sale  - 
Modernization  (Point  of  Sale  System),  the  Standard  Automated  Voucher 
Examination  System,  and  the  DeCA  Automatic  Coupons  System.  DoD  reported 
those  systems  as  mission  critical  to  the  Office  of  Management  and  Budget. 


Objectives 

The  Deputy  Secretary  of  Defense  requested  this  audit.  The  overall  audit 
objective  was  to  evaluate  whether  DeCA  adequately  planned  for  and  managed 
the  year  2000  risk  to  avoid  undue  disruption  to  its  mission.  Specifically,  we 
reviewed  the  year  2000  risk  assessments,  testing,  and  contingency  plans  for 
DeCA  mission-critical  systems  and  infrastructure  equipment.  See  Appendix  A 
for  a  discussion  of  the  audit  scope,  methodology  and  prior  audit  coverage. 
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Status  of  Year  2000  Compliance 

The  Defense  Commissary  Agency  was  making  progress  in  its  year  2000 
conversion,  but  more  could  be  done  to  minimize  risk.  The  Defense 
Commissary  Agency  target  certification  date  of  October  31,  1999,  for 
the  Defense  Commissary  Agency  Interim  Business  System  may  not  allow 
sufficient  time  to  correct  unexpected  errors,  changes,  and  delays  in 
solving  year  2000  issues.  In  addition,  as  of  April  9,  1999,  the  Defense 
Commissary  Agency  had  identified  the  compliance  status  of  only  1,822 
(55  percent)  of  its  3,285  vendor  models  of  infrastructure  equipment  that 
are  critical  to  the  operation  of  the  individual  commissaries  and 
administrative  offices.  Also,  the  contingency  plan  for  the  Defense 
Commissary  Agency  Interim  Business  System  was  inadequate,  and  the 
contingency  plan  for  the  infrastructure  equipment  was  incomplete. 

These  conditions  occurred  because  DeCA  did  not  focus  on  a  centralized 
management  strategy  described  in  the  DoD  Management  Plan  until 
May  1998;  cancelled  the  acquisition  of  the  replacement  system  for  the 
DeCA  Interim  Business  System;  and  decided  to  modernize  the  DeCA 
Interim  Business  System  before  making  it  year  2000  compliant.  DeCA 
also  experienced  contractual  disputes  and  a  lack  of  resource  availability. 

As  a  result,  the  delay  in  starting  the  renovation  of  DeCA  mission-critical 
systems  and  infrastructure  equipment  has  increased  risk.  The  DeCA 
must  therefore  intensively  manage  the  remaining  year  2000  conversion 
tasks  and  take  additional  action  to  minimize  risk. 


Identification  of  Year  2000  Deficiencies  and  Corrective 
Actions 


Using  the  DoD  definition  of  mission-critical,  DeCA  designated  the  Standard 
Automated  Voucher  Examination  System,  the  DeCA  Interim  Business  System, 
the  DeCA  Automatic  Coupons  System,  and  the  Point  of  Sale  System  as  mission 
critical.  The  systems  are  needed  for  DeCA  to  operate  the  commissaries 
(purchase,  stock,  and  sell  groceries  and  pay  vendors).  DeCA  had  certified  one 
of  the  systems,  the  Standard  Automated  Voucher  Examination  System,  as  year 
2000  compliant  before  DoD  defined  the  certification  guidelines.  Commendably, 
DeCA  decided  to  recertify  the  system  using  the  guidelines.  The  DeCA 
identified  the  deficiencies  and  necessary  fixes  needed  for  the  other  three 
mission-critical  systems  to  become  year  2000  compliant.  The  fixes  ranged  from 
rewriting  code  to  upgrading  software. 

In  addition  to  the  four  mission-critical  systems,  we  reviewed  the  DeCA  year 
2000  program  for  handling  infrastructure  equipment  that  is  critical  to  the 
operation  of  each  commissary  and  administrative  office. 
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Standard  Automated  Voucher  Examination  System.  The  Standard 
Automated  Voucher  Examination  System  provides  contracting,  bill  paying,  and 
accounting  support  for  DeCA.  The  DeCA  took  the  initiative  to  complete  the 
year  2000  repairs  to  the  Standard  Automated  Voucher  Examination  System  in 
October  1996  and  certified  it  as  year  2000  compliant.  However,  the 
certification  procedures  used  did  not  follow  those  subsequently  defined  in  the 
DoD  Management  Plan.  In  May  1998,  with  the  establishment  of  and  direction 
from  the  DeCA  Year  2000  Program  Office,  the  System  Manager  for  the 
Standard  Automated  Voucher  Examination  System  decided  to  recertify  the 
system  using  the  certification  process  defined  by  the  DoD  Management  Plan. 
Testing  started  in  March  1999,  which  was  the  earliest  opportunity  to  use  the 
developmental  test  bed. 


DeCA  Interim  Business  System.  The  DeCA  Interim  Business 
System  is  an  on-line,  interactive  inventory  management  system  that  supports 
most  business  functions  within  DeCA.  DeCA  used  two  methods  to  identify  the 
year  2000  deficiencies  within  the  DeCA  Interim  Business  System.  First,  DeCA 
rolled  over  the  date  on  the  system’s  test  platform  to  the  year  2000  and  processed 
transactions  for  approximately  2  months.  Second,  programmers  reviewed  the 
source  code  for  the  DeCA  Interim  Business  System  to  identify  any  that  were  not 
year  2000  compliant.  Using  the  two  methods,  DeCA  identified  58  programs 
that  were  noncompliant.  About  75  percent  of  the  58  programs  that  were  not 
year  2000  compliant  required  only  10  or  less  lines  of  source  code  to  be  rewritten 
to  be  compliant,  and  none  required  extensive  rewrites.  In  April  1999,  DeCA 
began  rewriting  the  source  code. 


DeCA  Automated  Coupons  System.  The  DeCA  Automated 
Coupons  System  tabulates  and  processes  manufacturer  coupons  redeemed  at  the 
commissaries.  With  the  assistance  of  the  contractor,  DeCA  identified  and 
rewrote  the  noncompliant  lines  of  source  code  and  tested  the  software  on  the 
operating  platform.  The  tests  showed  that  although  the  software  processed  date 
data  accurately,  the  operating  platform  was  not  year  2000  compliant;  therefore, 
the  system  needed  to  be  migrated  to  a  year  2000  compliant  operating  platform. 
The  DeCA  started  migrating  the  DeCA  Automated  Coupons  System  to  a 
year  2000  compliant  operating  platform  and  is  testing  the  system  on  the  new 
platform. 


Point  of  Sale  System.  The  commissaries  use  the  Point  of  Sale 
System  to  process  customer  purchases  and  capture  sales  and  financial  data.  The 
contractor  for  the  Point  of  Sale  System  identified  the  upgrades  needed  to  make 
the  system’s  hardware  and  software  year  2000  compliant.  DeCA  started  testing 
the  software  in  November  1998  to  ensure  that  it  met  all  functional  requirements. 
An  independent  testing  organization  also  tested  the  upgraded  software  for 
functionality.  When  DeCA  completes  the  testing  and  the  contractor  fixes  all  the 
software,  DeCA  will  start  to  deploy  the  software  and  hardware  upgrades  to  its 
294  commissaries,  4  regional  offices,  and  headquarters  offices. 
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In  addition  to  reviewing  the  four  mission-critical  systems,  we  reviewed  DeCA 
infrastructure  equipment  that  is  critical  to  the  operation  of  the  individual 
commissaries  and  administrative  offices.  DeCA  uses  the  infrastructure 
equipment  to  operate  the  commissaries  and  to  complete  administrative  duties. 
Examples  include  telephones,  refrigeration  monitoring,  meat  wrapping,  and  fire 
alarm  systems.  In  its  evaluation  of  year  2000  implications  for  infrastructure 
equipment,  DeCA  defined  16  categories  as  critical  to  the  operation  of  the 
commissaries.  In  November  1998,  DeCA  established  a  working  group  within 
the  Year  2000  Program  Office  to  organize  and  manage  the  infrastructure 
equipment  for  year  2000  compliance.  The  DeCA  conducted  a  100  percent, 
wall-to-wall  inventory  of  all  infrastructure  equipment  in  commissaries  and 
offices.  As  of  April  9,  1999,  the  inventory  showed  that  3,285  vendor  models  of 
infrastructure  equipment  fell  under  the  critical  categories.  Of  these,  DeCA 
knew  the  year  2000  compliance  status  of  only  1,822  vendor  models  of 
equipment  (55  percent).  The  compliance  status  of  the  remaining  1,463  vendor 
models  of  equipment  (45  percent)  was  unknown.  DeCA  plans  to  upgrade  and 
replace  all  noncompliant  equipment  by  September  30,  1999. 


Target  Dates  for  Completing  Year  2000  Solutions 


During  its  implementation  of  the  DoD  Management  Plan,  DeCA  developed  the 
DeCA  Year  2000  Management  Plan  (Plan),  dated  December  1998.  The  Plan 
determined  March  31,  1999,  as  the  target  completion  date  for  all  systems  within 
DeCA.  Thus,  it  was  implicitly  acknowledged  that  all  four  mission-critical 
systems  would  miss  the  December  31,  1998,  DoD  goal  and  be  termed  high  risk. 

r~ 

As  of  April  8,  1999,  three  of  the  DeCA  mission-critical  systems  were  in  the 
validation  phase;  the  DeCA  Interim  Business  System  was  in  the  renovation 
phase;  and  infrastructure  equipment  was  in  the  assessment  phase.  The  target 
dates  established  for  fixing,  testing,  and  deploying  year  2000  compliant  systems 
and  infrastructure  equipment  range  from  April  30,  1999,  to  October  31,  1999, 
which  may  not  allow  for  any  unexpected  errors,  changes,  or  delays  in  the  year 
2000  compliance  effort.  The  following  table  shows  the  specific  estimated 
completion  dates  for  the  four  mission-critical  systems  and  the  infrastructure 
equipment. 


Estimated  Compliance  Dates  for  DeCA  Mission- Critical  Systems 
and  Infrastructure  Equipment 


Mission-Critical  System 

Current  Phase 

Estimated 
Implementation/ 
Completion  Date 

DeCA  Automated 

Coupon  System 

Validation  Phase 

April  30,  1999 

DeCA  Interim 

Business  System 

Renovation  Phase 

October  31,  1999 

Point  of  Sale  System 

Validation  Phase 

August  26,  1999 

Standard  Automated 

Voucher  Examination 
System 

Validation  Phase 

May  25,  1999 

Infrastructure  Equipment 

Assessment  Phase 

September  30,  1999 

Reasons  for  Delays  in  the  DeCA  Year  2000  Effort 

Although  DeCA  had  been  planning  efforts  to  prepare  for  year  2000  since  1996, 
DeCA  did  not  focus  on  using  the  centralized  management  strategy  described  in 
the  DoD  Management  Plan  for  making  its  mission-critical  systems  and 
infrastructure  equipment  year  2000  compliant  until  May  1998.  DeCA 
management  had  not  directed  its  components  to  follow  the  centralized 
management  strategy.  Also,  each  system  had  a  unique  problem  that  further 
delayed  the  implementation  of  year  2000  solutions. 

Management  Strategy.  In  May  1998,  DeCA  established  the  Program  Office 
and  directed  that  system  managers  use  the  centralized  management  strategy 
defined  in  the  DoD  Management  Plan.  Before  it  established  the  Program 
Office,  DeCA  did  not  follow  the  guidance  in  the  DoD  Management  Plan.  This 
delay  hindered  DeCA  in  achieving  year  2000  compliance  for  its  mission-critical 
systems. 

Other  Issues.  In  addition  to  the  delay  incurred  by  not  complying  with  the  DoD 
Management  Plan,  the  following  system-unique  issues  further  delayed  DeCA 
efforts: 

•  canceling  a  replacement  system, 

•  deciding  to  modernize  systems  before  resolving  the  year  2000  issues, 

•  contractual  disputes,  and 

•  resource  availability. 
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DeCA  Interim  Business  System.  Delays  occurred  in  the  DeCA 
Interim  Business  System,  in  large  part,  because  DeCA  cancelled  the  acquisition 
of  the  Defense  Commissary  Information  System.  DeCA  was  confident  that  the 
Defense  Commissary  Information  System,  a  commercial  off-the-shelf  business 
system,  would  replace  the  DeCA  Interim  Business  System  and  resolve  the 
DeCA  year  2000  problem.  The  Defense  Commissary  Information  Systeqi  was 
to  mirror  the  commercial  grocery  sector  and  eliminate  the  increasing  cost  of 
hardware  maintenance  and  upgrades  on  a  proprietary  platform.  After  realizing 
that  it  could  not  change  its  business  practices  to  mirror  the  commercial  grocery 
store,  DeCA  shifted  the  focus  of  the  Defense  Commissary  Information  System 
to  a  software  development  program.  The  decision  adversely  affected  the 
program  cost  and,  by  mid- 1998,  the  system  was  2  years  behind  schedule  and  the 
projected  cost  to  complete  the  contract  was  almost  $100  million  over  the 
original  estimated  cost  of  $58  million.  In  May  1998,  the  Director,  DeCA, 
decided  to  discontinue  the  acquisition  of  the  Defense  Commissary  Information 
System. 

When  DeCA  cancelled  the  Defense  Commissary  Information  System,  it  decided 
to  eliminate  the  increasing  cost  of  hardware  maintenance  for  the  proprietary 
platform.  To  accomplish  this,  DeCA  decided  to  convert  the  existing  software  to 
make  the  DeCA  Interim  Business  System  to  a  new  “open  systems”  platform. 

The  DeCA  could  have  accelerated  DeCA  Interim  Business  System  year  2000 
compliance  by  making  the  changes  needed  to  the  58  programs  on  the  proprietary 
platform,  testing  and  certifying  the  system  as  year  2000  compliant  and  then, 
continuing  with  the  conversion  effort.  However,  DeCA  management  made  the 
conversion  its  highest  priority  to  establish  a  firm  baseline,  preferring  to  dedicate 
the  resources  to  the  conversion  effort.  Because  it  believed  that  year  2000  fixes 
would  be  minor  and  the  contractor  for  the  existing  hardware  and  operating 
system  would  not  certify  that  it  was  year  2000  compliant,  DeCA  decided  to 
modernize  the  system  before  making  it  year  2000  compliant. 


Point  of  Sale  System.  The  delay  in  the  Point  of  Sale  System  was 
caused  by  a  dispute  about  the  requirements  of  the  original  contract.  In 
February  1996,  DeCA  awarded  a  4-year  contract  to  NCR  to  procure  the  Point 
of  Sale  System  for  the  commissaries.  The  contract  had  four  1-year  options  and 
required  the  system  to  be  functional  at  the  time  of  installation,  which  would  be 
beyond  the  year  2000.  In  November  1997,  DeCA  informed  NCR  that  the 
contract  required  the  current  version  of  the  Point  of  Sale  System  software  and 
hardware  to  be  year  2000  compliant.  At  this  point,  a  legal  battle  ensued  for 
NCR  to  replace  the  software  and  hardware  with  year  2000  compliant  versions 
without  additional  cost  to  DeCA.  In  early  1998,  DeCA  and  NCR  reached  a 
consensus. 


DeCA  Automated  Coupons  System.  The  DeCA  Automated 
Coupons  System  was  delayed  because  the  necessary  technical  personnel  were 
assigned  to  the  DeCA  Bad  Check  System  to  make  changes  required  by  law. 
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Infrastructure  Equipment.  Shortage  of  available  personnel  was 
also  the  primary  reason  that  DeCA  delayed  its  efforts  to  inventory  infrastructure 
equipment  and  upgrade  or  replace  noncompliant  equipment. 


DeCA  Year  2000  System  Compliance  Risk 


DeCA  was  still  at  risk  for  year  2000  disruptions.  DeCA  maintained  that  the 
year  2000  compliance  changes  were  minor  and,  even  if  the  changes  were  not 
made,  the  business  functions  performed  by  the  DeCA  Interim  Business  System 
would  not  be  affected.  For  the  Point  of  Sale  System,  the  commissaries  could 
still  make  sales;  however,  they  could  not  execute  credit  and  debit  card 
transactions.  For  the  DeCA  Automated  Coupons  System,  the  commissaries 
would  manually  batch  and  process  manufacturer  coupons  for  redemption. 
However,  failure  of  infrastructure  equipment  would  affect  the  operations  of  the 
individual  commissaries.  Because  DeCA  had  not  completed  testing  of  the 
mission-critical  systems,  the  potential  year  2000  impact  on  its  operations 
remained  uncertain. 


Year  2000  Contingency  Plans 

The  DoD  Management  Plan  states  that  system-level  contingency  plans  should 
detail  the  procedures  necessary  to  restore  a  system  in  the  face  of  all  anticipated 
and  unanticipated  year  2000  disruptions.  The  contingency  plans  should  also 
provide  for  continuing  operations  when  the  support  from  a  single  system  or 
group  of  closely  related  systems  is  disrupted. 

DeCA  developed  adequate  contingency  plans  for  three  of  its  four  mission- 
critical  systems:  the  Point  of  Sale  System,  the  DeCA  Automated  Coupons 
System,  and  the  Standard  Automated  Voucher  Examination  System.  However, 
the  contingency  plan  for  the  DeCA  Interim  Business  System  was  inadequate. 
Specifically,  the  DeCA  Interim  Business  System  contingency  plan  addressed 
actions  to  take  if  DeCA  did  not  complete  the  conversion  of  the  DeCA  Interim 
Business  System  to  the  new  “open  systems”  platform;  it  did  not  address  the 
actions  needed  if  system  validation  is  not  completed  before  the  year  2000. 
Further,  the  DeCA  Interim  Business  System  contingency  plan  did  not  address 
the  actions  needed  if  the  system  is  believed  to  be  year  2000  compliant  but 
experiences  a  system  failure  at  the  start  of  year  2000.  In  addition  to  the  DeCA 
Interim  Business  System,  DeCA  did  not  complete  contingency  plans  for  each  of 
the  critical  infrastructure  equipment  categories. 

The  failure  of  DeCA  to  develop  complete  and  adequate  contingency  plans  for 
the  DeCA  Interim  Business  System  and  the  infrastructure  equipment  further 
increases  the  risk  of  mission-critical  systems  and  infrastructure  equipment  not 


7 


being  operational  after  December  31,  1999,  and  of  responsible  personnel  not 
knowing  what  to  do  when  faced  with  year  2000  failures. 

The  DeCA  system  managers  for  the  DeCA  Interim  Business  System  and 
infrastructure  equipment  were  in  the  process  of  updating  and  completing 
contingency  plans. 


Conclusion 


The  DeCA  was  making  progress  to  make  its  mission-critical  systems  and  critical 
infrastructure  equipment  year  2000  compliant.  However,  the  DeCA  year  2000 
conversion  strategy  is  not  without  significant  risk  and  sound  contingency 
planning  is  particularly  important.  DeCA  needs  to  intensively  manage  its 
high-risk  systems  and  take  additional  measures  to  minimize  risk. 


Recommendations  and  Management  Comments 

We  recommend  that  the  Director,  Defense  Commissary  Agency: 

1.  Expand  the  contingency  plan  for  the  DeCA  Interim  Business  System 
to  delineate  alternative  operational  procedures  that  will  be  used  to  perform 
its  functions  if  a  system  fails  as  a  result  of  the  year  2000. 

2.  Expand  the  contingency  plan  for  the  DeCA  Interim  Business  System 
to  delineate  the  operational  procedures  by  which  the  mission  or  function 
supported  by  the  system  will  be  continued  during  a  prolonged  disruption  of 
the  system. 

3.  Complete  contingency  plans  for  each  critical  infrastructure 
equipment  category  that  identifies  alternative  actions  to  be  taken  at  DeCA 
offices  and  commissaries  if  infrastructure  equipment  fails  as  a  result  of  the 
year  2000. 

4.  Test  the  contingency  plans  in  the  above  recommendations  to  ensure 
that  the  alternative  procedures  are  realistic  and  executable. 

Management  Comments.  The  Executive  Director  for  Support,  Defense 
Commissary  Agency  concurred  and  stated  that  DeCA  had  prepared  and 
expanded  the  contingency  plan  for  the  DeCA  Interim  Business  System.  The 
contingency  plan  delineates  alternative  operational  procedures  that  will  be  used 
to  perform  functions  if  a  system  fails  as  a  result  of  the  year  2000  and  ensure  the 
mission  or  functions  are  continued  during  a  prolonged  disruption  of  the  system. 
In  addition,  DeCA  is  in  the  process  of  developing  a  handbook  on  contingency 
plans  for  infrastructure  equipment  within  the  commissaries.  The  Executive 
Director  stated  that  DeCA  has  made  significant  progress  in  lowering  the 
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year  2000  risk  associated  with  evaluating  the  compliance  status  of  infrastructure 
equipment  that  are  critical  to  the  operation  of  the  individual  commissaries  and 
administrative  offices.  As  of  May  10,  1999,  DeCA  had  identified  the 
compliance  status  of  2,055  (79  percent)  of  2,840  vendor  models  of 
infrastructure  equipment.  (Note:  Subsequent  to  the  audit,  DeCA  revalidated  its 
infrastructure  equipment  inventory  and  identified  discrepancies  which  reduced 
the  total  inventory  of  vendor  models  from  the  3,285  previously  reported  to  the 
2,840  referenced  in  its  response.)  Further,  DeCA  is  currently  testing  or  will 
test  the  contingency  plans  to  ensure  that  the  alternative  procedures  are  realistic 
and  executable.  The  target  completion  date  for  all  ongoing  tasks  is 
October  1999. 
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Appendix  A.  Audit  Process 


This  report  is  one  in  a  series  being  issued  by  the  Inspector  General,  DoD,  in 
accordance  with  an  informal  partnership  with  the  Chief  Information  Officer, 
DoD,  to  monitor  DoD  efforts  to  address  the  year  2000  computing  challenge. 
For  a  list  of  audit  projects  addressing  the  issue,  see  the  year  2000  web  page  on 
the  IGnet  at  http: // www. ignet.  gov. 


Scope 


We  reviewed  and  evaluated  the  efforts  of  DeCA  to  resolve  its  year  2000  issues. 
Specifically,  we  reviewed  the  steps  taken  by  the  system  manager  for  each  of  the 
four  systems  that  DeCA  identified  as  mission  critical.  We  compared  the  DeCA 
year  2000  compliance  management  efforts  with  DoD  policies  and  procedures, 
including  documentation  and  reporting  requirements,  to  assess  whether  the 
efforts  were  adequately  conducted  to  avoid  undue  year  2000  disruption.  We 
obtained  documentation  dated  from  May  1996  through  April  1999,  including  the 
DeCA  inventory  of  mission-critical  systems,  contingency  plans,  and  test  plans, 
to  assess  efforts  to  avoid  undue  year  2000  disruptions.  We  also  reviewed  the 
year  2000  efforts  for  critical  infrastructure  equipment  because  a  year  2000 
failure  in  this  area  would  directly  influence  the  operation  of  the  commissaries, 
as  well  as  the  Headquarters,  DeCA. 

DoD- Wide  Corporate-Level  Government  Performance  and  Results  Act 
Goals.  In  response  to  the  Government  Performance  and  Results  Act,  DoD  has 
established  6  DoD-wide  performance  level  objectives  and  14  goals  to  meet  these 
objectives.  This  report  pertains  to  the  achievement  of  the  following  objective 
and  goal: 

Objective:  Maintain  highly  ready  joint  forces  to  perform  the  full 
spectrum  of  military  activities.  Goal:  Recruit  and  retain  well-qualified 
military  and  well-qualified  military  and  civilian  personnel.  (DoD-5.2) 

DoD  Functional  Area  Reform  Goals.  Most  major  DoD  functional  areas  have 
also  established  performance  improvement  reform  objectives  and  goals.  This 
report  pertains  to  the  achievement  of  the  following  functional  area  objectives 
and  goals: 

Information  Technology  Management  Functional  Area. 

•  Objective:  Become  a  mission  partner.  Goal:  Serve  mission 
information  users  as  customers.  (ITM-1.2) 
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•  Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Modernize  and  integrate  Defense  information  infrastructure. 

(ITM-2.2) 

•  Objective:  Provide  services  that  satisfy  customer  information  needs. 
Goal:  Upgrade  technology  base.  (ITM-2.3) 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  Department  of  Defense.  This  report 
provides  coverage  of  the  year  2000  conversion  high-risk  area. 


Methodology 

Use  of  Technical  Assistance  and  Computer-Processed  Data.  We  did  not  use 
technical  assistance  or  computer-generated  data  to  perform  this  audit. 

Audit  Period,  Standards,  and  Locations.  We  performed  this  economy  and 
efficiency  audit  from  December  9,  1998,  through  April  9,  1999,  in  accordance 
with  auditing  standards  issued  by  the  Comptroller  General  of  the  United  States, 
as  implemented  by  the  Inspector  General,  DoD. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  and 
organizations  within  DoD  and  contractors  for  the  systems  at  DeCA.  Further 
details  are  available  on  request. 

Management  Control  Program.  We  did  not  review  the  management  control 
program  related  to  the  overall  audit  objective  because  DoD  recognized  the 
year  2000  computing  problem  as  a  material  management  control  weakness  area 
in  the  FY  1997  and  FY  1998  Annual  Statements  of  Assurance. 


Summary  of  Prior  Coverage 


The  General  Accounting  Office  and  the  Inspector  General,  DoD,  have 
conducted  multiple  reviews  related  to  year  2000  issues.  General  Accounting 
Office  reports  can  be  accessed  over  the  Internet  at  http://www.gao.gov. 
Inspector  General,  DoD,  reports  can  be  accessed  at  http://www.dodig.osd.mil. 
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Appendix  B.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Director,  Defense  Logistics  Studies  Information  Exchange 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Under  Secretary  of  Defense  for  Personnel  and  Readiness 
Assistant  Secretary  of  Defense  (Command,  Control,  Communications  and 
Intelligence) 

Deputy  Assistant  Secretary  of  Defense  (Command,  Control,  Communications,  and 
Intelligence,  Surveillance,  Reconnaissance,  and  Space  Systems) 

Deputy  Chief  Information  Officer  and  Deputy  Assistant  Secretary  of  Defense  (Chief 
Information  Officer  Policy  and  Implementation) 

Principal  Director  for  Year  2000 
Assistant  Secretary  of  Defense  (Public  Affairs) 


Joint  Staff 

Director,  Joint  Staff 


Department  of  the  Army 

Chief  Information  Officer,  Department  of  the  Army 
Inspector  General,  Department  of  the  Army 
Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy,  (Financial  Management  and  Comptroller) 

Chief  Information  Officer,  Department  of  the  Navy 

Inspector  General,  Department  of  the  Navy 

Auditor  General,  Department  of  the  Navy 

Inspector  General,  Marine  Corps 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force,  (Financial  Management  and  Comptroller) 
Chief  Information  Officer,  Department  of  the  Air  Force 
Inspector  General,  Department  of  the  Air  Force 
Auditor  General,  Air  Force  Audit  Agency 
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Unified  Commands 


Commander  in  Chief,  U.S.  European  Command 
Commander  in  Chief,  U.S.  Pacific  Command 
Commander  in  Chief,  U.S.  Atlantic  Command 
Commander  in  Chief,  U.S.  Southern  Command 
Commander  in  Chief,  U.S.  Central  Command 

Other  Defense  Organizations 

Director,  Defense  Commissary  Agency 
Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Information  Systems  Agency 

Inspector  General,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 


Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 

Office  of  Information  and  Regulatory  Affairs 
General  Accounting  Office 

National  Security  and  International  Affairs  Division 
Technical  Information  Center 

Director,  Defense  Information  and  Financial  Management  Systems,  Accounting  and 
Information  Management  Division 


Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  of  Appropriations 

Senate  Committee  on  Armed  Services 

Senate  Committee  on  Governmental  Affairs 

Senate  Special  Committee  on  Year  2000  Technology  Problem 

House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform 

House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
House  Subcommittee  on  Technology,  Committee  on  Science 
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Defense  Commissary  Agency 
Comments 


DEFENSE  COMMISSARY  AGENCY 
HEA00UARTER8 
1900  E  AVENUE 

FORT  LEE,  VIRGINIA  99I01-190D 


MEMORANDUM  FOR  INSPECTOR  GENERAL,  ACQUISITION  MANAGEMENT 

DIRECTORATE,  400  ARMY  NAVY  DRIVE,  ARLINGTON,  VA 
22202-2884 

SUBJECT:  Audit  Report  on  the  Status  of  Year  2000  Compliance  at 
the  Defense  Commissary  Agency  Operations  (Project  No. 
9AL-0083) 

Reference:  DoDIG  Memorandum,  April  20,  1999,  SAB 

Attached  is  the  DeCA  reply  to  subject  report  as  requested 
in  referenced  memorandum.  If  you  have  any  questions,  please 
contact  Mr.  Ben  Mikell  at  (804)  734-8103. 

$hr  CROSBY  H.  JOHNSON 

Executive  Director  for  Support 

Attachment: 

As  Stated 
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Final  Repori 
Reference 


DEFENSE  COMMISSARY  AGENCY  REPLY 

SUBJECT:  Audit  Report  on  the  Status  of  Year  2000  Compliance  at 
the  Defense  Commissary  Agency  Operations  (Project  No. 
9AL-0083) 

ADDITIONAL  COMMENTS  RELATED  TO  RECOMMENDTATIONS  1  AND  2 

The  audit  report  states  "DeCA  confidence  that  year  2000 
certification  will  be  accomplished  before  December  31,  1999,  and 
that  operations  will  not  be  affected  if  year  2000  changes  are 
not  made,  has  resulted  in  DeCA  not  deeming  the  development  of 
complete  and  adequate  contingency  plans  for  all  of  its  mission- 
critical  systems  and  infrastructure  equipment  to  be  a  high 
priority'1 . 

In  actuality,  the  DeCA  Interim  Business  System  (DIBS)  2000 
is  the  contingency  plan  for  the  Defense  Commissary  Information 
System  (DCIS) .  The  Air  Force  Standard  Systems  Center  awarded  a 
contract  on  DeCA' s  behalf  to  Computer  Sciences  Corporation  in 
July  1995  for  a  Y2K  compliant,  modernized  grocery  business 
system  (DCIS) .  In  May  1998,  DeCA  decided  to  let  the  contract 
expire  because  DCIS  could  not  be  completely  fielded  by  Year  2000 
and  because  DCIS  was  competing  with  DeCA' s  Y2K  initiatives  for 
available  IT  resources.  DoDIG  has  audited  DCIS  (Inspector 
General,  DoD  Direction  from  the  Report  of  the  House  Committee  on 
National  Security  for  H.  R.  3616,  National  Defense  Authorization 
Act  for  Fiscal  Year  1999,  DoDIG  Project  Number  9AL-5030.)  May 
1998  was  a  pivotal  date  for  DeCA' s  Y2K  efforts  because  the 
uncertainties  surrounding  DCIS  functionality  and  schedule  were 
finally  resolved.  Once  the  DCIS  contract  expired,  DeCA  could 
focus  dollar  and  human  resources  on  Y2K. 

Confidence  in  certification  and  a  belief  that  operations 
could  continue  were  not  the  reasons  that  DeCA  had  not  developed 
DIBS  contingency  plans .  Having  invoked  the  primary  contingency 
plan,  we  had  only  enough  resources  to  ensure  it  was  well 
underway  before  developing  secondary  and  tertiary  contingency 
plans. 

The  audit  report  states  "Because  it  believed  that  year  2000 
fixes  would  be  minor  and  the  contractor  for  the  existing 
hardware  and  operating  system  would  not  certify  that  it  was  year 
2000  compliant,  DeCA  decided  to  modernize  the  system  before 
making  it  year  2000  compliant". 
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Deleted 
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While  waiting  for  DCIS,  DeCA  spent  minimal  dollars  to  keep 
the  current  DIBS  baseline  operational.  But  as  with  any  software 
production  baseline,  DeCA  had  to  change  the  system  to 
accommodate  changes  in  operational  procedures .  The  system  could 
not  be  put  in  moratorium  as  mission-critical  system  change 
packages  had  to  be  released.  DeCA  had  identified  the  required 
changes  for  DIBS  Y2K  in  1996  and  received  notification  from  the 
hardware  vendor  in  1997  that  he  would  not  migrate  the  hardware 
and  operating  system  for  Y2K  compliance.  The  mean-time-between- 
failures  rate  for  the  computers  was  increasing  steadily 
beginning  in  1997.  Additional  machines,  replacement  parts,  and 
knowledgeable  field  engineers  to  support  the  obsolete  technology 
were  becoming  scarce.  The  machines  were  seven  years  old  by  May 
1998  when  the  DCIS  decision  was  made  and  were  beginning  to  fail 
sporadically.  DeCA  assessed  the  risk  of  total  hardware  failure 
between  May  1998  and  January  2000  against  making  additional 
changes  to  the  software  baseline  required  by  January  2000. 

Given  the  criticality  of  the  production  system,  we  determined 
that  the  risk  of  continuing  with  the  obsolete  hardware  and 
further  agitating  the  production  baseline  with  Y2K  changes  was 
greater  than  that  associated  with  making  Y2K  changes  on  a  stable 
baseline. 

RECOMMENDATION  1.  Expand  the  contingency  plan  for  the  DeCA 
Interim  Business  System  to  delineate  alternative  operational 
procedures  that  will  be  used  to  perform  its  functions  if  a 
system  fails  as  a  result  of  the  Year  2000. 

DeCA  REPLY.  CONCUR.  This  recommendation  has  been  completed. 
DeCA  has  prepared  and  expanded  the  contingency  plan  for  the  DIBS 
to  delineate  alternative  operational  procedures  that  will  be 
used  to  perform  its  functions  if  a  system  fails  as  a  result  of 
the  Year  2000. 

RECOMMENDATION  2.  Expand  the  contingency  plan  for  the  DeCA 
Interim  Business  System  to  delineate  the  operational  procedures 
by  which  the  mission  or  function  supported  by  the  system  will  be 
continued  during  the  prolonged  disruption  of  the  system. 

DeCA  REPLY,  CONCUR.  This  recommendation  has  been  completed. 
DeCA  has  prepared  and  expanded  the  contingency  plan  for  the  DIBS 
to  delineate  the  operational  procedures  by  which  the  mission  or 
function  supported  by  the  system  will  be  continued  during  a 
prolonged  disruption  of  the  system. 
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ADDITIONAL  COMMENTS  RELATED  TO  RECOMMENDATION  3 

The  audit  report  states  "As  of  April  9,  1999,  the  Defense 
Commissary  Agency  had  identified  the  compliance  status  of  only 
1822  (55  percent)  of  3,285  vendor  models  of  infrastructure 
equipment  that  are  critical  to  the  operation  of  the  individual 
commissaries  and  administrative  offices”.  From  March  to  date, 
the  DeCA  Y2K  PMO  has  intensely  managed  equipment  and  has 
successfully  lowered  the  increased  risk  identified.  As  of  May 
10,  1999,  DeCA  had  identified  the  compliance  status  of  2,055 
(78.73  percent)  of  2,840  vendor  models  of  infrastructure 
equipment  that  are  critical  to  the  operation  of  the  individual 
commissaries  and  administrative  offices. 

Reference  Page  5,  Estimated  Compliance  Dates,  for 
Infrastructure  Equipment,  first  paragraph.  Infrastructure 
Current  Phase  only  states  Assessment  Phase.  The  Implementation 
Phase  is  being  executed  concurrently  with  the  Assessment  Phase 
at  this  time,  in  order  to  meet  the  estimated  completion  date  of 
September  30,  1999. 

RECOMMENDATION  3.  Complete  contingency  plans  for  each  critical 
infrastructure  equipment  category  that  identifies  alternative 
actions  to  be  taken  at  DeCA  offices  and  commissaries  if 
infrastructure  equipment  fails  as  a  result  of  the  Year  2000. 

DeCA  REPLY.  CONCUR.  DeCA  staffed  a  draft  handbook  DeCAH  40-5, 
Non-It  Store  Contingency  Plans  on  April  22,  1999,  and  requested 
comments  from  offices  and  commissaries  on  the  critical 
infrastructure  equipment  categories  by  May  7,  1999.  Responses 
have  now  been  received  and  are  being  incorporated  into  the  draft 
document.  The  target  completion  date  is  October  1999. 

ADDITIONAL  COMMENTS  RELATED  TO  RECOMMENDATION  4 

The  contingency  plan  for  DIBS  is  a  two-level  approach. 

Level  one,  currently  being  tested,  is  running  year  2000 
transactions  on  the  current  baseline.  Level  two  is  the  business 
continuance  or  continuity  of  operations  plan  in  effect  today. 

If  there  is  a  failure  in  any  piece  of  the  supply  chain,  store- 
specific  orders  are  pre-staged  on  each  region  processor,  the 
Headquarters  processor,  and  the  supplier's  processor.  A  phone 
call  communicates  the  emergency  order  for  the  store  to  the 
supplier  so  that  the  shelves  can  be  replenished. 

Non-IT  infrastructure  equipment  contingency  plans  are 
currently  being  tested  at  several  commissaries  in  each  region. 
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RECOMMENDATION  4.  Test  the  contingency  plans  in  the  above 
recommendations  to  ensure  that  the  alternative  procedures  are 
realistic  and  executable. 

DeCA  REPLY.  CONCUR.  DeCA  is  currently  testing  or  will  test  the 
contingency  plans  as  recommended  to  ensure  that  the  alternative 
procedures  are  realistic  and  executable.  The  target  completion 
date  is  October  1999. 
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